Robust concurrency control in main-memory DBMS: What main memory giveth, the application taketh away

Modern systems with large main memories and massively parallel processors have inspired many new high-performance OLTP systems [2–4,6], often referred to as main memory DBMS (MMDBMS). These systems leverage spacious main memory to fit the whole working set in DRAM with streamlined, memory-friendly data structures; further, optimizations for multicore and multi-socket hardware allow a much higher level of parallelism compared to conventional database systems. With disk overheads and delays removed, transaction laten-cies drop precipitously and worker threads can usually execute transactions to completion without interruption. The result is a welcome reduction in contention and less pressure on whatever concurrency control (CC) scheme might be in place. Meanwhile, database workloads are evolving to become increasingly heterogeneous , blending the gap between transaction and analytical processing. This trend is at least partly enabled by the improved concurrency and reduced contention offered by MMDBMS. Mixed workloads have two significant impacts on CC, however. First, the read/write ratio increases from 2:1 (e.g. TPC-C) to 10:1 or higher [1], usually by increasing the number of reads as the number of writes remains stable. Second, workloads frequently include some fraction of large transactions that are read-mostly rather than read-only, a trend reflected in the TPC-E [5] benchmark. Unfortunately, both of these workload properties mean an increase in effective concurrency control footprints, and increased pressure on the CC scheme. As is usually the case, it appears that our workloads stand ready to absorb any and all concurrency gains the MMDBMS has to offer. In this talk, we argue that the shift to heterogenous workloads means effective and robust CC schemes will become increasingly important for main-memory DBMS going forward. A growing body of research shows that the CC schemes currently in vogue with MMDBMS are not robust under contention, particularly when short write-intensive transactions coexist with longer read-mostly transactions. For example, the two most common families of approaches can be loosely classified as two-phase locking (2PL) and optimistic concurrency control (OCC). 2PL is common in traditional disk-oriented systems, and is often criticized because of high overheads, its policy of blocking transactions (leading to deadlocks and other scheduling problems), and a tendency to lock up (performance crash) once the aggregate transactional footprint grows too large (a state quickly attained when large transactions enter the system). OCC, on the other hand, never